Dashboard Overview

This dashboard compares coaching bot versions based on session metrics, annotations, and user engagement.

Generated: 2025-11-07 14:04:43

Dashboard Filters

to
Enter one participant ID per line. Leave empty to show all participants.
Note: Date and participant filters work for the Session Volume chart. Refrigerator filter applies to all graphs and tables. Participant filter for tables requires dashboard regeneration. Outlier filtering affects the progression line graph and median words/messages tables.

Summary Metrics by Version

Coaching Bot Version # Sessions # Annotated Sessions Refrigeration Examples (%) Median Human Words per Session Average Session Rating
Control bot 3355 0 0.0% 37.0 4.78
Coaching bot V3 888 80 0.0% 124.0 4.71
Coaching bot V4 892 172 33.1% 106.0 4.85
Coaching bot V5 1291 62 37.1% 152.0 4.53
Coaching bot V6 1210 3 33.3% 122.0 4.65
Total (All Versions) 7636 317 24.9% 122.0 4.71

Summary Metrics - All Versions vs Refrigerator Examples

Category # Sessions # Annotated Sessions Refrigeration Examples (%) Median Human Words per Session Average Session Rating
All Versions 7636 317 24.9% 122.0 4.71
Refrigerator Example Sessions Only 81 81 100.0% 174.0 4.81

Refrigerator Example Rate by Method and Version

Method Control botCoaching bot V3Coaching bot V4Coaching bot V5Coaching bot V6 All Versions
Scenario--100.0%100.0%-100.0%
Microlearning--72.3%100.0%-86.2%
Microlearning vaccines---66.7%-66.7%
Motivational interviewing--62.5%100.0%100.0%87.5%
Visit check in--100.0%100.0%-100.0%
Unknown------
Total (All Methods)--83.7%93.3%100.0%88.1%

Average FLW Score by Method and Version

Method Control botCoaching bot V3Coaching bot V4Coaching bot V5Coaching bot V6 All Versions
Scenario-4.774.774.464.594.65
Microlearning--4.844.554.634.67
Microlearning vaccines---4.474.654.56
Motivational interviewing--4.874.694.704.75
Visit check in-4.634.944.644.664.72
Unknown4.79---5.004.89
Total (All Methods)4.794.704.864.564.704.71
Rating Collection Statistics:
• Rating Questions: 83.3% of sessions (6384 out of 7666) contain rating questions
• Actual Ratings: 81.1% of sessions (6214 out of 7666) have extractable ratings

Median Number of Participant Messages per Session by Method and Version

Method Control botCoaching bot V3Coaching bot V4Coaching bot V5Coaching bot V6 All Versions
Scenario-12.011.014.012.012.0
Microlearning--13.014.513.013.0
Microlearning vaccines---13.013.013.0
Motivational interviewing--16.519.017.017.0
Visit check in-13.015.018.018.016.5
Unknown9.03.03.0-10.06.0
Total (All Methods)9.012.013.014.513.013.0

Median User Words per Session by Method and Version

Method Control botCoaching bot V3Coaching bot V4Coaching bot V5Coaching bot V6 All Versions
Scenario-127.0100.0151.0132.0129.5
Microlearning--87.5122.090.090.0
Microlearning vaccines---93.590.091.8
Motivational interviewing--107.5167.5134.0134.0
Visit check in-118.5130.5209.5177.0153.8
Unknown37.08.013.0-50.025.0
Total (All Methods)37.0118.5100.0151.0111.0110.6

Median User Words - Session Progression Analysis

Volume of Session per Coach Version

Session Count by Method and Version

Method Control botCoaching bot V3Coaching bot V4Coaching bot V5Coaching bot V6 All Versions
Scenario-6372405532481678
Microlearning--232186231649
Microlearning vaccines---176237413
Motivational interviewing--208190246644
Visit check in6242206186243883
Unknown334996-53369
Total (All Methods)3355888892129112107636

Definitions

Version Definitions

This dashboard compares five bot categories based on experiment IDs and version ranges:

  • Control bot: Experiment ID: 1027993a-40c9-4484-a5fb-5c7e034dadcd (All versions)
  • Coaching bot V3: Experiment ID: e2b4855f-8550-47ff-87d2-d92018676ff3 (All versions)
  • Coaching bot V4: Experiment ID: b7621271-da98-459f-9f9b-f68335d09ad4 (Version 13 and above)
  • Coaching bot V5: Experiment ID: 5d8be75e-03ff-4e3a-ab6a-e0aff6580986 (Version 1 to 4)
  • Coaching bot V6: Experiment ID: 5d8be75e-03ff-4e3a-ab6a-e0aff6580986 (Version 5 and above)
Coaching Methods

Coaching methods are detected through a 3-tier approach:

  • Scenario: Roleplay and scenario-based coaching
  • Microlearning: Quiz and microlearning sessions
  • Microlearning vaccines: Vaccine-specific quiz sessions
  • Motivational interviewing: Motivational interview techniques
  • Visit check in: Home visit debrief sessions
Average FLW Score by Method and Version

This metric shows the average session rating (1-5 scale) grouped by coaching method and bot version.

  • Calculation: Average of user ratings for sessions ending with "How useful did you find this coaching session? Please rate it from 1 to 5"
  • Rating Scale: 1 (not useful) to 5 (very useful)
  • Method Detection: Based on session tags (coach_method_*) or message content analysis
  • Version Detection: Based on experiment ID and version tags from last message
  • Purpose: Identify whether coaching methods receive higher ratings with bot evolution
Median Number of Participant Messages per Session by Method and Version

This metric shows the median number of messages participants send per session, grouped by coaching method and bot version.

  • Calculation: Median count of user messages within a session
  • Method Detection: Based on session tags (coach_method_*) or message content analysis
  • Version Detection: Based on experiment ID and version tags from last message
  • Message Count: Total number of user messages in a session
Median User Words per Session by Method and Version

This metric shows the median number of words users type per session, grouped by coaching method and bot version.

  • Calculation: Median word count of all user messages within a session
  • Method Detection: Based on session tags (coach_method_*) or message content analysis
  • Version Detection: Based on experiment ID and version tags from last message
  • Word Count: Total words across all user messages in a session
  • Outlier Filtering: Optional checkbox to exclude sessions with >50 messages or >1000 words
Session Numbering:

For line graph analysis, sessions are numbered chronologically per participant:

  • Participant ID: Based on participant.identifier field
  • Chronological Order: Sorted by session created_at timestamp
  • Session Number: Position in participant's session sequence (1st, 2nd, 3rd, etc.)